Validating airspace Analyzer Metrics for En Route ATC Sector Complexity

The MITRE Corporation’s Center for Advanced Aviation System Development (CAASD) has developed a tool, airspaceAnalyzer, to measure sector complexity. It does so by performing automated air traffic control—separating traffic while respecting aircraft performance and restrictions on altitude and miles-intrail. It outputs a suite of raw metrics, derived from characteristics of these resolutions. We performed experiments first to calibrate, then to validate, the model. To define a “correct” value for sector complexity against which to calibrate and validate, we enlisted a team of experts (retired Certified Professional Controllers). The experts performed air traffic control for en route sectors in Indianapolis Center in a realistic simulation environment. They provided periodic assessments of the sector’s complexity on a scale from 1 (minimum) to 7 (maximum). We averaged the assessments of all participating experts at a particular time to define the correct answer for that time. We calibrated airspaceAnalyzer to output a metric that matched as closely as possible the “correct” answers. Then, we performed validation experiments on a new set of four scenarios, which featured realistic, but somewhat different, traffic flows and restrictions. The validation set involved three Indianapolis sectors (including one not used in calibration) and seven experts (including three not used in calibration). Assessments were obtained from each expert at 20-30 times during each scenario, for a total of 101 data points. Using its calibrated metric, airspaceAnalyzer predicts the correct answer with a root mean square error of at most 0.687 points on the 1-7 scale. That it could match the human experts this well is evidence for the validity of airspaceAnalyzer as a predictor of sector complexity.